Robust frame size error detection and recovery mechanism to minimize frame loss for camera input sub-systems

ABSTRACT

An image data frame is received from an external source. An error concealment operation is performed on the received image data frame in response to determining that a first frame size of the received image data frame is erroneous. The first frame size of the image data frame is determined to be erroneous based on at least one frame synchronization signal associated with the image data frame. An image processing operation is performed on the received image data frame on which the error concealment operation has been performed, thereby enabling an image processing module to perform the image processing operation without entering into a deadlock state and thereby prevent a host processor from having to execute hardware resets of deadlocked modules.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. application Ser. No.16/745,589, filed Jan. 17, 2020, which claims priority to U.S.Provisional Application No. 62/956,379, filed Jan. 2, 2020, and U.S.Provisional Application No. 62/956,975, filed Jan. 3, 2020, each ofwhich is hereby incorporated by reference.

BACKGROUND

Autonomous driving is one of the world's most challenging computationalproblems. Different functions related to autonomously driving ormaneuvering a vehicle can be implemented using advanced driverassistance systems (ADAS) that rely on sensor equipment that providesdata of a host vehicle's environment. Such sensor equipment may includeRADAR (radio detection and ranging), LIDAR (light detection andranging), cameras for imaging, and the like. Implemented ADAS functionsmay range from distance sensing and parking assistance up tosophisticated ADAS functions like cruise control, lane change assist,collision mitigation, emergency braking, fully autonomous driving, andthe like. Very large amounts of data from image sensors, RADARs, LIDARs,and HD-Maps must be processed to generate commands to control a vehiclesafely and comfortably in real-time in such ADAS systems. Thischallenging task requires one or more dedicated computing devices thatare energy-efficient and low-power, run complex high-performancesoftware, and rely on breakthroughs in artificial intelligence, machinelearning, deep learning, and computer vision. Such computing devices maybe implemented as energy-efficient and space-efficient System-on-a-Chips(SoCs), which may be integrated into a flexible, expandable platformthat enables a wide-range of autonomous vehicles.

Typical computing devices (e.g., SOCs) include multiple internalcomponents that exchange data. For example, a typical processor mayinclude multiple functional blocks such as image signal processors,display engines, and/or media encoders that exchange and process imagedata. Errors in an incoming continuous stream of image data processed byimage signal processor modules on SoCs may not only cause corrupt pixeldata that renders image frames unacceptable for use in safety criticalapplications (e.g., autonomous driving, computer vision) but the faultyframes may also cause other types of issues (e.g., deadlock conditions)in an image processing pipeline which require a hardware reset of one ormore modules of the SoC to correct the problem. Using a reset to restartthe processing modules has undesirable side effects and often results inloss of additional image frames which runs counter to the highperformance, high reliability, and low-latency requirements of safetycritical applications. For example, it may be unacceptable to havemultiple image data frame losses in an ADAS system that relies on theimage data frames for autonomous driving.

SUMMARY

The following presents a simplified summary of the disclosed subjectmatter in order to provide a basic understanding of some aspects of thisdisclosure. This summary is not an exhaustive overview of the technologydisclosed herein. It is not intended to identify key or criticalelements of the disclosure or to delineate the scope of the disclosure.Its sole purpose is to present some concepts in a simplified form as aprelude to the more detailed description that follows.

In one example, an image processing device comprises an image datareceiver, an error handler, and an image processor, in which the imagedata receiver includes an error detector. The image data receiver isconfigured to receive an image data frame, and the error detector isconfigured to detect whether a size of the image data frame iserroneous. The error handler is coupled to, and configured to receivefrom, the error detector a current image data frame. The error handleris configured to perform frame level error operations on the currentimage data frame to generate a corrected image frame. The frame levelerror operations include detecting a size violation with respect to thecurrent image data frame based on a comparison of received first,second, third, and fourth signals and expected first, second, third, andfourth signals, in which the expected first signal is active while theerror handler receives a first pixel of an image data frame, theexpected second signal is active while the error handler receives a lastpixel of an image data frame, the expected third signal is active whilethe error handler receives a first pixel of each of a plurality of linesincluding the first line of an image data frame, and the expected fourthsignal is active while the error handler receives a last pixel of eachof the plurality of lines including the last line of an image dataframe. The image processor is coupled to receive from the error handlerthe corrected image data frame.

In another example, an image processing device comprises an image datareceiver that includes an error detector. The image data receiver isconfigured to receive an image data frame, and the error detector isconfigured to detect whether a size of the image data frame iserroneous. The image processing device further comprises an errorhandler coupled to, and configured to receive from, the error detector acurrent image data frame. The error handler is configured to performline level error operations on the current image data frame, includingfetching data of lines of the current image data frame line-by-line. Foreach fetched line, the error handler is configured to compare multiplereceived synchronization signals to expected synchronization signals todetermine whether at least one of the multiple synchronization signalsexhibits an improper transition or is not received, compare an actuallength of the line to a reference length, and perform line errorhandling when the actual length of the line is determined to bedifferent than the reference length based on the compare operation togenerate a corrected line, the line error handling including concealingthe line size error from a downstream image processor coupled to theerror handler.

In still other examples, the functionality of at least some of any ofthe aforementioned imaging processing devices may be implemented as amethod or embodied in computer executable program code and stored in anon-transitory storage device.

BRIEF DESCRIPTION OF THE DRAWINGS

For a detailed description of various examples, reference will now bemade to the accompanying drawings in which:

FIG. 1 shows a block diagram of an image processing system according toone or more embodiments.

FIG. 2 shows a timing chart illustrating frame-level error detection andhandling operation performed by the image processing system according toone or more embodiments.

FIG. 3 shows another timing chart illustrating frame-level errordetection and handling operation performed by the image processingsystem according to one or more embodiments.

FIG. 4 shows a flowchart of an image processing method for frame-levelerror detection and handling that may be performed by the imageprocessing system according to one or more embodiments.

FIG. 5 illustrates line-level error detection and handling operationperformed by the image processing system according to one or moreembodiments.

FIG. 6 shows a flowchart of an image processing method for line-levelerror detection and handling that may be performed by the imageprocessing system according to one or more embodiments.

FIG. 7 shows an illustrative simplified block diagram of a computingsystem according to one or more embodiments.

DETAILED DESCRIPTION

This disclosure pertains to detecting and efficiently handling (e.g.,concealing) image frame size and/or synchronization signal protocolerrors to minimize frame loss in safety critical applications.Techniques disclosed herein look to avoid deadlock situations in modulesor units of an image processing pipeline caused by mishandling ofunexpected missing (i.e., undersize) and/or extra (i.e., oversize) pixeldata of an image data frame or mishandling caused by an otherwisecorrupt image frame (e.g., protocol error). When a continuous stream ofimage frames is being received from a source (e.g., image sensor) and isbeing processed in an image processing pipeline real-time, on-the-fly,one or more image processing units of the pipeline may be configured toreceive the image frames, each having a specific frame size (referencesize) and/or adhering to a predetermined protocol (e.g., synchronizationsignal protocol).

Various types of errors may be introduced into the bitstream at thesource, during transmission, or at the receiver modules of the pipeline,and may cause data corruption of an image frame so that the corruptimage frame no longer adheres to its size and/or protocol configurationsas expected by the receiver. That is, uncorrected image sensor dataand/or transmission errors in incoming image data bitstream can cause acaptured frame to have the wrong frame size due to missing pixel imagedata and/or faulty frame synchronization information of the input streamand due to data loss following an internal overflow from system trafficoverload.

Processing of such a corrupt image frame at the receiver may causevarious types of issues in the image processing pipeline, some of whichmay be more fatal than others. For example, a corrupted image frame thatis directly stored into memory and that has erroneously become largerthan its expected size (e.g., oversized frame) may cause memory overflowerrors from extra writes to the memory caused by the oversized inputframe. As another example, a corrupted image frame that is subject toon-the-fly processing by one or more image processing units of the imageprocessing pipeline may unexpectedly cause the receiver image processingunit may enter into an unknown state (e.g., lockup state, deadlockstate, and the like) because of the frame size and/or protocol beingdifferent from that expected by the receiver.

While corrupt pixel image data only results in corruption of the imagedata that is stored in memory, frame size or protocol errors duringon-the-fly processing can lead to many side effects, including memorycorruption or the image processing units (and sub-units) of the imageprocessing pipeline entering into a deadlock state due to the faultyframe. Once the module has entered into the deadlock state, the onlyoption to resolve the deadlock for a host processor implementing theimage processing pipeline may be to execute a hardware reset. However,using hardware resets to restart image processing units which areprocessing incoming image data in real-time applications require thehost processor to go through software interrupt processing handling.This interrupt processing handling may, in turn, lead to loss ofmultiple frames that are being received continuously due to the timerequired for executing the reset of the stalled components (andsub-components) of the image processing pipeline. Such loss of framesmay be unacceptable in safety critical applications like ADAS,infotainment, imaging, computer vision, and the like, that rely on thecontinuous stream of image data to make real-time decisions.

To prevent image processing units of the pipeline from entering into thedeadlock state, and thereby prevent the host processor from having toexecute hardware resets, techniques disclosed herein look to efficientlyand quickly handle faulty (e.g., illegal or erroneous size) incomingframes so that even if the image data of the frame is corrupted to thepoint where the pixel image data of the frame may be no longer useablefor subsequent processes, an error detection and handling moduleimplemented in the image processing pipeline may detect the frame sizeerror of the faulty incoming frame, and execute appropriate errorhandling steps to quickly conceal the size error during real-timeprocessing so that the ‘size-corrected’ frame may pass through the imageprocessing pipeline without causing any deadlock states.

For example, when an undersized input frame is detected, the errorhandling module may operate to maintain the full input frame size asexpected by a downstream image processing unit by generating missingpixels and lines prescribed for the current frame (i.e., full-size frameemulation for downstream processing module). Any remaining incoming datafor the faulty undersized frame may be discarded by the handler whilefilling up the remainder of the frame (pixels and lines) with ‘dummy’data. As another example, when an oversized input frame is detected, theerror handler may discard remaining incoming data and initialize to waitfor start of a next incoming frame. In either erroneous size case, asoftware interrupt may be issued to the host processor to identify thefaulty frame (and provide details of the fault to software layer, e.g.,oversized, undersized, protocol error, and the like) that is saved intomemory. The host may then implement high-level error handling options(e.g., discard faulty frame and repeat good frame, and the like).

Other embodiments of the error handling module may perform errorhandling operations on the stream of image frames on a ‘line-by-line’basis. That is, the error handling module may perform processing foreach erroneously sized (undersized or oversized) input line of the frameby either generating missing pixel data of the current faulty line ordiscarding the remainder of incoming pixel data of the oversized line,and then handle the next line of pixel data of the frame independentlyof the previous line. Thus, line-based (line-level) error handling mayreplicate missing pixel data only within the current line and the nextline is handled as a normal line (with any applicable error handlingbased on detected size error). This scheme may minimize full frame dropmerely because of a localized error condition (e.g., error in only oneline of an otherwise ‘normal’ frame). The software interrupt to the hostprocessor in the case of line-based error handling may then includeadditional error information that is reported to assist the software inmaking a decision of whether or not to completely reject the erroneousframe based on error location information, region of interestinformation, and a pixel confidence factor indicating the number ofpixels/lines that had to be replicated (e.g., number of dummypixels/lines) in the erroneous frame.

FIG. 1 shows a block diagram of image processing system 100 according toone or more embodiments. Image processing system 100 may be implementedfor a variety of different applications that require real-timeprocessing of an incoming stream of image data (e.g., ADAS,infotainment, imaging, computer vision applications). As shown in FIG. 1, control unit 105 (e.g., image processing device), which may beimplemented as a system-on-a-chip (SoC), may be used for receiving acontinuous stream of image data from multiple image data sourcesutilizing different interfaces. In the embodiment shown in FIG. 1 ,control unit 105 may receive video streams from external sourcesincluding a digital parallel video stream received via video interface110 and connected to an external video decoder 115 or another processor.Control unit 105 may also receive a continuous stream of image dataframes via image data receiver 120 that is connected over a serialinterface to image data transmitter 125.

In one embodiment, the serial interface connection to transmitter 125provides high-speed image data transfer via a standardized communicationinterface. For example, transmitter 125 and receiver 120 may beconnected via camera serial interface (CSI) which is a specification ofthe mobile industry processor interface (MIPI) alliance standard. CSIdefines an interface between transmitter (e.g., image sensor, camera,and the like) and control unit 105 (e.g., host processor, SoC, and thelike). The image data stream received by receiver 120 may be acontinuous sequence of image frames (e.g., received at 30 frames persecond) that have a given resolution (e.g., 1 or 2 megapixels).

Image data received via video interface 110 may be at a lower speedrelative to the high-speed CSI-MIPI-compliant serial image data transfervia receiver 120. Data input via interface 110 may be saved into systemmemory 130 via system interconnect 135 and memory interface 140. Thevideo data may also be displayed on display 145 via display interface150. On the other hand, data received via receiver 120 when control unit105 is used for certain applications (e.g., automotive) where lowprocessing latency is critical may be transferred to an image processingpipeline implemented using one or more image processing units 155 beforethe stream of processed image data frames is written out to systemmemory 130 for further processing or displayed on display panel 145. Forexample, image processing unit 155 may be an image signal processor(ISP) or hardware accelerator (HWA) module that converts RAW input imagedata to one of the internal processing color space formals (e.g., YUV orRGB color space formats) and/or performs additional on-the-fly colorimage processing. Although FIG. 1 illustrates the image processingpipeline being implemented using only one image processing unit 155,this may not necessarily be the case. More than one image processingunits 155 may form the image processing pipeline on control unit 105.

When implemented as a CSI-MIPI-compliant interface, the serial imagedata transfer via receiver 120 may include multiple channels of imagedata streams transferred over the same physical interface. Receiver 120of control unit 105 may extract pixel image data (e.g., payload data)out of the received continuous stream of image data and sync signalinformation (e.g., start/end of frame, start/end of each line of theframe, and the like) from the input bitstream and transfer the receivedinput stream of image data frames to either system memory 130 via directmemory access (DMA) controller 160 or to the connected image processingunit 155 (e.g., HWA module or ISP module) of the image processingpipeline.

While the bitstream is continuously transmitted from transmitter 125 toreceiver 120 and subsequently stored into system memory 130 directly orpost-processing via the image processing pipeline, various errorconditions may be introduced into the input image data at various pointsalong the signal chain. For example, due to various fault conditions,errors may possibly be introduced at the image sensor source (e.g.,transmitter 125), on the transport bitstream between transmitter 125 andreceiver 120, at receiver 120, and/or at one or more image processingunits 155 of the image processing pipeline downstream of receiver 120.Possible sources of error conditions include glitches in an image sensoror image data transmitter 125 causing error in pixel data and/or syncsignals at the source, transmission errors from transmitter 125 tocontrol unit 105 not detected/corrected by receiver 120, and errorsintroduced by receiver 120 due to wrong user configuration of framedimension that does not match received frame size, unsupported formatdetection, and undetected design issues. Other sources of error in thebitstream may include an overflow condition in the receiver 120resulting in pixel data and/or sync signal information loss and randomtransient bit corruption due to, e.g., electromagnetic interference inthe signal path causing erroneous input frame handling at receiver 120.

When pixel image data has an error (e.g., faulty pixel values), even ifthe corresponding image frame may be unusable for display or for makingautonomous driving decisions, the data can still be passed down to theimage processing pipeline for processing without causing fatal deadlockconditions. On the other hand, when the frame data has the wrong linesize or fame size or other protocol error, the subsequent imageprocessing pipeline may be unable to handle the corresponding faultyimage frame. Failure to detect and/or correct the error conditions,particularly on the line/frame size and sync information, can lead toproblems like memory corruption (e.g., overwrites) from a frame with asize error (e.g., oversized frame) being written directly into memory130. Failure to detect and/or correct the error conditions can also leadto frame corruption due to misaligned sync signals and in certain cases,cause deadlock conditions in the image processing pipeline due tomishandling of unexpected missing and/or extra pixels/lines of a frame.

Error detection and correction mechanisms such as cyclic redundancycheck (CRC) and error correction code (ECC) may be implemented on thetransport stream between transmitter 125 and receiver 120 to check forpayload data and single-bit error correction and 2-bit error detectionin the packet header to protect integrity of transmitted data. However,CRC and ECC do not provide protection from frame size and protocolerrors which may originate from the upstream image capture module andwhich may cause fatal deadlock conditions in the image processingpipeline. The frame size and protocol errors may also originate in amodule that is downstream of receiver 120. Error detection module 121and error handling module 156 may detect and appropriately handle suchsize or protocol errors.

Image data receiver 120 extracts and sends image data frames inreal-time to memory 130 and/or to downstream image processing unit 155.Image data receiver 120 may include error detection module 121 whichimplements a frame size error detection mechanism based on actual framesizes (e.g., second frame size) against user configured frame referencesizes. In particular, error detection module 121 may perform errordetection by checking line width and frame height of input image frame(actual size) against the expected frame size (e.g., reference size) asper the user configuration to detect size deviations. Error detectionmodule 121 may further provide memory protection by discarding extrapixel data of one or more lines when an oversize condition (e.g., extrapixel data in one or more lines or extra lines in one frame) isdetected. By discarding the data, error detection module 121 protectsmemory 130 from extra writes which may otherwise be caused by anoversized input frame.

When error detection module 121 detects an error, the software issues anerror interrupt to notify a host processor to handle the resulting errorframe. For example, the host processor may mark the frame which may besaved into memory 130 as having a size error. While error detectionmodule 121 performs frame size error detection and memory protection foroversize frames, error detection module 121 may still pass frames withshort lines (e.g., missing pixel data in one or more lines; short line)or less than the standard number of lines (e.g., missing one or morelines of the frame; short frame) or frames with other protocol errors tothe downstream image processing unit 155 for image processing. Such aframe passed downstream to image processing unit 155 may cause imageprocessing unit 155 to enter into an unknown state leading to a lockupcondition. That is, image processing unit 155 configured to receiveincoming image data streams may expect exact frame sizes to be input andmay lack error handling when an input frame having an unexpected framesize is received. Deviations in the frame size or protocolconfigurations may lead to output corruptions or even fatal deadlockconditions which require a hardware reset of unit 155 and any downstreammodules to resolve the deadlock. For example, the deviations can leaveimage processing unit 155 of the image processing pipeline in a deadlockstate in which unit 155 is waiting for more data to complete the currentimage data frame processing or in an error state in which unit 155 failsto appropriately handle extra (unexpected) image pixel data of one ormore lines of the frame.

To resolve such a deadlock state and restore normal processing of unit155 for a next image frame of the continuously captured stream of imageframes passing through the image pipeline in real-time, the hostprocessor may have to execute a hardware reset to flush affected one ormore processing units 155 and restart the pipeline. Further, thedeadlock state of processing unit 155 may be detected only when anothererror (e.g., overflow error) is detected and reported later. This mayinevitably result in an error recovery with multiple frame losses whichis unacceptable for safety critical automotive applications.

To efficiently handle faulty frames passing through the image processingpipeline, image processing unit 155 may include error handling module156. Error handling module 156 may be configured to perform frame sizeerror detection and error concealment/handling operations prior toperforming image processing operations at image processing unit 155. Forexample, error handling module 156 may detect frame sync signal protocolviolations and frame size errors for each incoming image frame. Errorhandling module 156 may further perform frame size error concealmentoperations for detected faulty frames (or lines) and corresponding errorreporting and restart operations.

To detect faulty frames having an illegal frame or line size, errorhandling module 156 may perform size error checks on incoming image datastream of frames by comparing current measured frame size (e.g., firstframe size) against reference frame and line sizes (e.g., referencesize) provided by the user (i.e., user configuration). For example,error handling module 156 may detect line width errors includingdetecting a short line in which a given input line of a frame has dataof less than the expected number of active pixels and detecting a longline in which a given input line of a frame has data of more than theexpected number of active pixels. Error handling module 156 may furtherdetect frame height errors including detecting a short frame in whichthe current input frame has data of less than the expected number ofactive lines and detecting a long frame in which the current input framehas data of more than the expected number of active lines. When errorhandling module 156 detects a frame size error (e.g., long line, shortline, long frame, short frame, and the like), error handling module 156generates an interrupt to indicate the frame size error to the hostprocessor and goes into error handling mode (e.g., error concealmentoperation).

To detect frame size errors, error handling module 156 may look forframe sync signal protocol violations in incoming frames by utilizing aset of frame synchronization markers (frame synchronization signals)that mark start and end of lines and start and end of frames when imagedata frames are transferred between receiver 120 and image processingunit 155. Synchronization markers that may be defined and used include:

Signal name Default State Description VP.PCLK 0 Pixel clock. VP.VS 0Active during the 1st pixel of the frame VP.VE 0 Active during the lastpixel of the frame VP.HS 0 Active during the 1st pixel of any line VP.HE0 Active during the last pixel of any line VP_STALL 1 Video port stalledwhen 1 (optional)

When the continuous stream of image data frames is being received byimage processing unit 155, error handling module 156 may check eachincoming frame to ensure that the synchronization markers or signalstransition properly for each normal frame input. Error handling module156 may utilize an illegal line or frame transition detected based onthe received frame synchronization markers as an early indicator of aframe size error in the current input frame. In one embodiment, errorhandling module 156 may perform the following checks using the framesync signals or markers to detect frame sync signal protocol violationsfor incoming frames from receiver 120 that are input to image processingunit 155.

VE without HE Frame End (VE) is received without the Line End (HE) VSwithout HS Frame Start (VS) is received without the Line Start (HS)VS-VS check Frame Start (VS) is not paired with a Frame End (VE) HS-HScheck Line Start (HS) is not paired with a Line End (HE) HE-HE checkLine End (HE) is followed by a second Line End (HE) VE-VE check FrameEnd (VE) is followed by a second Frame End (VE)

When one or more of the above frame sync signal protocol violations aredetected, error handling module 156 generates an interrupt and reportsthe detected violations (e.g., protocol violation or error). A protocolerror (e.g., missing sync signal, misaligned sync signal, and the like)generally requires the frame to be discarded. To ensure the frame sizeis maintained at expected levels (reference size) for downstreammodules, error handling module 156 may further use the sync signals todetect illegal frame sizes. That is, the sync signals are used fordiagnosing frame size errors and reported to the host processor. Upondetecting frame size errors based on frame sync signals, error handlingmodule 156 goes into error handling mode to perform quick errorconcealment operations to prevent deadlock state of image processingunit 155 or beyond. Image processing unit 155 expects data of an exactnumber of pixels and lines for each received input image frame to workproperly (i.e., normal operation with no errors). Further, DMAcontroller 165 (and task controller) responsible for handling datatransfer between one or more image processing units 155 of the imageprocessing pipeline and system memory 130 may rely on image processingunit 155 to output each processed frame with an exact (expected as peruser configuration) frame size. When a frame size error is detected,error handling module 156 immediately switches to error recovery mode toperform undersized or oversized input frame error handling (errorconcealment operation). FIGS. 2 and 3 illustrate undersized input framehandling performed by error handling module 156 upon detection of anillegal frame size for the current frame.

As shown by way of the timing chart 200 in FIG. 2 , when error handlingmodule 156 detects an undersized input frame (e.g., short frame, shortline) while processing input Frame 1 (1—error detected) due to earlierthan expected receipt of frame synchronization marker VE indicating endof the Frame 1 (e.g., first frame or line size less than reference frameor line size), error handling module 156 goes into error handling mode(error concealment operation) for the current frame, discards anyremaining input data for current frame received from receiver 120(2—input data from receiver discarded; e.g., a remainder of the imagedata frame discarded) and generates ‘dummy’ data to complete the currentFrame 1 (3—short frame correction). Thus, in the error handling mode,error handling module 156 operates to maintain the full (or expected)input frame size for Frame 1 for processing by image processing unit155.

Error handling module 156 generates a complete frame for imageprocessing unit 155 by generating data of the missing pixels and linesprescribed for the current frame. Since the frame is deemed an errorframe, error handling module 156 may repeat image data of the last knowngood pixel to complete the frame. In other embodiments, other data fromthe frame or data other than current frame data may be used to generatethe dummy data. Generation of the dummy data is performed at fullprocessing clock speed, which is typically much faster than incomingdata which is coming from receiver 120 at pixel clock rate (VP.PCLK). Asa result, the error handler can complete the current frame, which isdeemed to be an error or faulty frame, as quickly as possible and beready for the next frame start (Frame 2) at the next correspondinginitialization signal (4—input data from receiver checked for VS signalflag and next frame captured normally; and 5—Frame 3 captured).

As shown in the exemplary timing chart 200 of FIG. 2 , by performingshort frame error correction as per the error handling and concealmentoperations of error handling module 156, reception of a ‘normal’ sizedframe at image processing unit 155 can be assured (even if the normalsized frame has bad pixel data), and as a result, image processing unit155 can be prevented from entering into any unknown or deadlock state.In FIG. 2 , if not for the undersized input frame error handling oferror handling module 156, image processing unit 155 may receive shortFrame 1 and enter into a deadlock state which would require the hostprocessor to perform software interrupt processing handling and executea hardware reset of image processing unit 155 to restart the unit andany sub-units. Such additional steps may cause delay which would lead toloss of additional one or more image frames (e.g., Frame 2, Frame 3, andso on) from the continuous stream of input image frames.

The timing chart shown in FIG. 3 illustrates an alternate scenario ofshort frame error handling (undersized input frame handling; errorconcealment operation) by error handling module 156 in one or moreembodiments. While the current error frame is being completed byperforming the error concealment operation and insertion of dummy data,error handling module 156 may discard the input from receiver 120 toflush any remaining error frame data. As shown in FIG. 3 , and as in thecase of the timing chart of FIG. 2 , when error handling module 156detects an undersized input frame (e.g., short frame, short line) whileprocessing input Frame 1 (1—Error Detected) due to earlier than expectedreceipt of frame synchronization marker VE indicating end of the Frame1, error handling module 156 goes into error handling mode for thecurrent frame, discards any remaining input data for current framereceived from receiver 120 (2—input data from receiver discarded) andgenerates ‘dummy’ data to complete the current Frame 1 (3—short framecorrection).

In such a scenario, if reception of new frame (Frame 2) starts on thecontinuous image data stream received from receiver 120 while flushingof current Frame 1 is still active, frame synchronization marker VSindicating start of the Frame 2 may arrive before error handling module156 initializes DMA and gets ready for start of the Frame 2 (4—inputframe 2 skip detected). In this case, Frame 2 may get skipped as welland error handling module 156 maintains the error handling mode anddiscards any remaining input data for current frame 2 received fromreceiver 120 and issues an interrupt to notify the host controller ofthe “extra” frame (i.e., Frame 2) loss during the error recovery. In anormal case, when performing error handling and concealment operationsfor a short frame, this “extra” frame loss will not happen due to avertical blanking period between incoming frames and due to much slowerpixel clock rates (i.e., rates at which new frames come in) as comparedto the full processing clock speed at which the error frame is discardedand dummy data generated.

Returning to FIG. 1 , when error handling module 156 detects anoversized input frame (e.g., long frame, line line) while processing thecurrent input frame received from receiver 120 (e.g., first frame orline size greater than reference frame or line size; a number of pixelsfor which data is received by error handling module 156 for at least oneof a plurality of lines of the image data frame is greater than areference number of pixels per line for the image data frame (longline); a number of lines for which data is received by error handlingmodule 156 for the image data frame is greater than a reference numberof lines of the image data frame (long frame)), error handling module156 goes into error handling mode to perform oversized input framehandling (error concealment operation) by discarding all remainingincoming streams of the current oversized image frame and going intoerror frame quick completion mode by generating the remaining pixel/linedata to complete the oversized frame as quickly as possible with dummydata. Since the frame is deemed an error frame, error handling module156 may repeat image data of the last known good pixel to complete theframe according to its normal expected size. In other embodiments, otherdata from the frame or data other than current frame data may be used togenerate the dummy data and complete the frame.

Generation of the dummy data is performed at full processing clockspeed, which is typically much faster than incoming data which is comingfrom receiver 120 at pixel clock rate (VP.PCLK). As a result of theerror handling operation performed for the oversized frame, a completeframe with a normal, expected size can be provided to image processingunit 155. Waiting to finish receiving the oversized frame and thensimply discarding data of the extra pixels and lines may take an unknowntime period. Also, since error conditions in the remainder of the imagedata to be received for the current erroneous oversized frame areunknown, waiting to finish receiving the oversized frame may cause thesystem to enter unknown states.

Once the error handler determines the current frame to be a faultyoversized frame, by completing the current oversized frame as quickly aspossible (e.g., by discarding incoming data of current frame and quicklygenerating dummy data to fill the frame), error handling module 156enables the shortest possible completion time of the current faultyframe so that error handling module 156 can complete initializing DMAand get ready for start of new frame before arrival of framesynchronization marker VS indicating start of the next frame. The errorconcealment operation of error handling module 156, that enables imageprocessing unit 155 to complete the current frame with the correct framesize, prevents any deadlock conditions in the control logic in the restof image processing unit 155 or in other downstream modules of the imageprocessing pipeline and enables seamless error recovery with minimalframe loss and without a hardware (sub-system) reset from a frame sizeerror.

FIG. 4 shows a flowchart of image processing method 400 that may beperformed by image processing system 100 according to one or moreembodiments. As shown in FIG. 4 , method 400 begins at block 405 whenimage data receiver 120 of control unit 105 receives an image dataframe. As explained previously, receiver 120 may receive a continuousstream of image data frames from an external source (e.g., image datatransmitter 125) consistent with a predetermined communication standard(e.g., CSI-MIPI-standard). At block 410, error detection module 121 ofreceiver 120 may perform an error detection operation on the receivedimage data frame by checking line width of one or more lines of theinput image frame and frame height of the frame (actual received size;second frame size) against the expected line size and frame size as perthe user configuration (reference size) to detect size deviations. If aframe size error is detected by error detection module 121 at block 410(YES at block 410), operation proceeds to block 415 where errordetection module 121 may issue an error interrupt (e.g., softwareinterrupt) to notify the host processor that the current received imageframe has a frame size error. The host processor may then performsoftware interrupt processing handling based on the received interruptto implement high-level error handling options (e.g., discard errorframe).

At block 420, error detection module 121 checks to determine if thecurrent frame with the frame size error detected at block 410 is anoversized frame. That is, error detection module 121 determines if datacorresponding to current image frame received by receiver 120 includesdata for more than the prescribed number of pixels of a line or data ofmore than the prescribed number of lines of the frame. If errordetection module 121 detects the frame to be oversized (YES at block420), at block 425, error detection module 121 discards the data of theextra pixels and/or lines. For example, error detection module 121 mayflush the remainder of the data received for the current image frameonce data of the prescribed number of pixels for the prescribed numberof lines has been received. Thus, at block 425, error detection module121 provides memory protection by discarding extra pixel data of one ormore lines when an oversize condition is detected. At block 425, if theframe is not an oversized frame (NO at block 420) (e.g., frame is ashort or undersized frame in which image data of one or more pixels ofone or more lines is missing; i.e., short frame or short lines), errordetection module 121 may simply pass the undersized frame to thedownstream module (e.g., error handling module 156, image processingunit 155) for further processing.

If no frame size error is detected by error detection module 121 atblock 410 (NO at block 410), or if error detection module 121 detectsthe frame is not oversized (NO at block 420), operation proceeds toblock 430 where receiver 120 (or error detection module 121) determineswhether the current image frame that has been subject to error detectionand memory protection is to be processed on-the-fly, in real-time by theimage processing pipeline implemented on control unit 105. That is, atblock 430, receiver 120 may route the received image frame directly toDMA controller 160 for storage on system memory 130 (NO at block 430;block 460) if the frame is not to be subject to processing at the imageprocessing pipeline. Alternately, receiver 120 may route the receivedimage frame to error handling module 156 for image processing by imageprocessing unit 155 (YES at block 430) if it is determined at block 430that the frame is to be processed at the image processing pipeline.

If the frame is to be processed on-the-fly, method 400 proceeds to block435 where the image frame is transmitted to the image processingpipeline that is implemented on control unit 105. That is, at block 435,receiver 120 (or error detection module 121) transmits the currentreceived image frame to error handling module 156 which in turntransmits the frame after error detection and concealment operations toone or more image processing units 155 constituting the image processingpipeline.

Method 400 then proceeds to block 440 where error handling module 156,that has received the current image frame from receiver 120, detectswhether the current received frame violates any of the frame sync signalprotocols. At block 440, error handling module 156 utilizes thepreviously described frame synchronization markers that are activeduring the beginning and end of each line, and beginning and end of eachframe to check whether any of the sync signals received at module 156are missing or misaligned (e.g., VE without HE, VS without HS, VS-VScheck, and the like). If error handling module 156 detects at block 440any violation of the frame sync signal protocols (YES at block 440),error handling module 156 issues an interrupt to the host processorindicating the protocol error for the current frame (block 442).Protocol errors generally result in the frame being discarded by thehost processor during subsequent processing. In order to preventdownstream modules from entering into unknown states and causingdeadlock conditions, at block 444, error handling module 156 furtherchecks if the current frame has an illegal frame size. At block 444,error handling module 156 checks the incoming video stream againstreference sizes to detect line width errors (e.g., long line, shortline) and frame height errors (e.g., long frame, short frame) based onthe synchronization markers.

If error handling module 156 detects at block 444 an illegal frame sizebased on the received sync signals and the user configured referencesizes (YES at block 444), error handling module 156 goes into errorhandling mode to quickly conceal the size error and maintain the framesizes as they are expected by downstream modules. This also preventssubsequent frame loss by avoiding deadlock conditions in the imageprocessing pipeline. In the error handling mode, error handling module156 performs undersized input frame handling (block 445A) or oversizedinput frame handling (block 445B), based on whether the frame detectedwith the illegal frame size at block 444 is undersized (e.g., shortframe, short line) or oversized (e.g., long frame, long line).

At block 450, error handling module 156 issues an interrupt to the hostprocessor indicating the frame size error for the current faulty framefor which the error handling and concealment operation was performed.After the error concealment operation, the faulty frame (whose framesize is now the expected size) is passed to the downstream module of theimage processing pipeline. If, on the other hand, error handling module156 does not detect any illegal frame size for the current frame (NO atblock 444), error handling module 156 transmits the error-free framedownstream to the image processing pipeline (e.g., image processing unit155) for image processing.

At block 455, the current frame is transmitted to the downstream moduleof the image processing pipeline (e.g., image processing unit 155) forimage processing. Since the frame transmitted at block 455 and receivedby image processing unit 155 is of the correct size (i.e., size asexpected by modules of the pipeline), deadlock conditions in the controllogic of the pipeline can be prevented and seamless error recovery isenabled without requiring a hardware reset, thereby minimizing frameloss which is imperative in safety critical applications (e.g., ADAS,computer vision). At block 460, the image frame subject to processing bythe image processing pipeline is saved to system memory 130. Method 400then proceeds to block 465 so that operations of blocks 405-465 arerepeatedly performed for each incoming image frame that is received bycontrol unit 105 from external image data sources (e.g., transmitter125).

The disclosure in FIGS. 1-4 is related to frame-based (or frame-level)error handling and concealment in which once a size error is detected atany point in a given image frame by error handling module 156, theremainder of the erroneous frame is discarded and filled with dummy dataand the interrupt is issued to notify the host processor of theerroneous frame. That is, in frame-based (or frame-level) error handlingas described above, once an error is detected in a given frame, theframe is marked as a faulty frame and the whole frame may be flushed(e.g., if the size error is in the first line of the frame), even ifonly a small portion of the frame has the error. In other words, inframe-based error handling, data of all remaining frame pixels followingthe first error detection is flushed even in the case of a localizederror condition. For example, pixel loss due to a momentary FIFOoverflow may lead to a single short line in an otherwise good frame.However, under frame-based error handling, the remainder of the framepixel data may be flushed and replaced with dummy data when the singleshort line is detected. In case the single short line occurs at the endof the frame (e.g., in the last line of the frame), the frame may stillbe discarded based on the interrupt to the host processor marking theframe as a faulty frame. However, it may be unnecessary or wasteful todiscard the whole frame based on a localized error condition.

In the embodiment shown in FIGS. 5 and 6 , the error handling andconcealment operation performed by the error handling module may beperformed on a line-by-line basis (i.e., line-based or line-level errorhandling) to minimize full frame drops (i.e., discarding the full frame)for localized error conditions. Line-based error handling replicatesdata for missing pixels only within the current line of the currentframe in which the error has been detected (e.g., based on correspondingframe sync signal protocol violation) and the next line of the currentframe is handled as a normal line, with the same error handling andconcealment operations, if applicable. The line-based error handlingmodule may then perform error information reporting that providesadditional information (e.g., error location information, frameconfidence value or factor information, and the like) to the hostprocessor along with the software interrupt to aid the processor inmaking a frame rejection decision (e.g., do not reject the frame if theerror is localized and is otherwise outside of a region of interest ofthe frame). For example, a status register associated with the softwareinterrupt may indicate the additional information regarding the type oferror (e.g., undersized frame or line, oversized frame or line, errorlocation information and the lie). Protocol errors (e.g., sync signalmisalignment) detected based on sync signal protocol violations arefatal and generally the frame is rendered useless. But in case oflocalized frame size errors (e.g., frames with illegal frame size), itmay be possible to use the frame (i.e., choose not to discard the faultyframe) despite the error, if the software determines (based on the errorinformation read by the software from additional registers associatedwith the error interrupt) that the region of interest corresponding tothe frame with the localized frame size error is error-free (or haserroneous pixel data lower than a predetermined threshold).

The line-based error handling module may keep track of error locations(e.g., which lines of the current frame are faulty) and report the errorregion or location information to the host when issuing the errorinterrupt. The host can then use the information to determine whether touse or discard the current frame data. For example, if the pixels orlines with the size error are outside the vertical and/or horizontalregion of interest area, then the error can be ignored. The line-basederror handling module can also keep track of the number of pixels and/orlines of the current frame that had to be replicated and may report theerror count information to the host when issuing the error interrupt.The module may also provide a confidence value (e.g., frame confidencelevel or number) to the host with the interrupt based on the number ofpixels and/or lines of the current frame that had to be replicated. Thehost can then use this information to better determine whether or not toreject the current faulty image frame completely.

As shown in FIG. 5 , the line-level error handling module (not shown)performs error detection and handling/concealment operations on aline-by-line basis. When the error handler detects a short line at linen+1 based on corresponding received line synchronization signals, pixelimage data, and reference line size, the line-level error handlingmodule performs line-level error detection and handling (undersizedinput line error handling; error concealment operation) operations forline n+1 of the current frame. In the example of FIG. 5 , when theline-level error handling module detects an undersized input frame(e.g., short line n+1) while processing the current input due to earlierthan expected receipt of frame synchronization marker HE indicating endof line n+1, line-level error handling module goes into error handlingmode for the current line n+1, and generates ‘dummy’ data to completethe current line n+1 with corresponding frame synchronization signals.Thus, in the line-level error handling mode, the error handler operatesto maintain the full (or expected) input line size for line n+1 forprocessing by downstream modules. In one embodiment, the line-levelerror handling module may repeat image data of the last known good pixelof line n+1 to complete the line. In other embodiments, other data fromthe line n+1 or data other than current line or frame may be used togenerate the dummy data. Generation of the dummy data is performed atfull processing clock speed, which is typically much faster thanincoming data which is coming from the receiver at pixel clock rate(VP.PCLK). As a result, the error handler can complete the current linen+1, which is deemed to be an error or faulty line, as quickly aspossible and be ready for the next line n+2.

After completing error processing and handling for short line n+1 inFIG. 1 , the line-level error handler may begin processing the next linen+2 of the current frame as a normal line, with the same error handlingand concealment operations, if applicable. In the case of frame-basederror handling, when error handling module 156 detects the short line atline n+1, all remaining lines starting from line n+1 (including linen+2) may be flushed and filled with dummy data. On the other hand, inthe case of line-level error handling, the next line is treated as anormal line, thereby minimizing discarding of ‘good’ pixel data as inthe case of a frame with a localized error (e.g., error in only one or afew lines).

Returning to FIG. 5 , when the error handler detects a long line at linen+m based on corresponding received line synchronization signals, pixelimage data, and the reference line size, the line-level error handlingmodule performs line-level error detection and handling operations(oversized input line error handling; error concealment operation) forline n+m of the current frame in which it discards all remainingincoming streams of the current oversized image line n+m, and again goesinto error frame quick completion mode by generating the remaining pixeldata to complete the oversized current line n+m as quickly as possiblewith dummy data. The error handler may repeat image data of the lastknown good pixel to complete the line n+m according to its normalexpected size. In other embodiments, other data from the line n+m ordata other than current line or frame may be used to generate the dummydata and complete the line n+m.

As a result of the error handling operation performed for the oversizedline n+m, a complete line with a normal, expected size can be providedto the downstream module. Waiting to finish receiving the oversized linen+m and then discarding data of the extra pixels may take an unknowntime period. Also, since error conditions in the remainder of the linen+m image data to be received for the current erroneous line areunknown, waiting to finish receiving the oversized line data may causethe system to enter unknown error states. Once the error handlerdetermines the current line n+m to be a faulty oversized line, bycompleting the current oversized line as quickly as possible (e.g., bydiscarding incoming data of current line and quickly generating dummydata to fill the line), the error handler enables the shortest possiblecompletion time of the current line so that it can get ready for startof new line n+m+1 of the current frame and receive the line n+m+1without error.

Further, as shown in FIG. 5 , the line-level error handling module mayreport the error pixel count (e.g., number of pixels that had to bereplicated for the short line n+1, number of pixels that had to bediscarded for the long line n+m+1), and the error location or region(e.g., error start line n+1, and error end line n+m+1) within thecurrent frame when issuing the interrupt to the host processor regardingthe current faulty frame. The handler may similarly perform processingfor subsequent lines of the current frame, with any error detection andhandling for each line, as needed.

FIG. 6 shows a flowchart of image processing method 600 that may beperformed by an image processing system including the line-based errorhandling module according to one or more embodiments. Only those stepsof line-level error handling method 600 that are different from theframe-level error handling method 400 of FIG. 4 are shown in FIG. 6 ,and described below in detail. Steps similar to those in frame-levelerror handling method 400 of FIG. 4 are omitted. In at least someimplementations, both the frame level size error checking and theline-based error recovery/concealment described with respect FIGS. 4 and6 are performed concurrently.

As shown in FIG. 6 , at block 630, the line-level error handling module,that has received the current image frame from the receiver of controlunit 105, fetches data of pixel lines of the current frame one line at atime. For the line fetched at block 630, the line-level error handlingmodule at block 640 detects whether the current received line of thecurrent frame violates any of the frame sync signal protocols. At block640, as explained previously, the error handler utilizes framesynchronization markers that are active during beginning and end of eachline to check whether the sync signals are missing or misaligned (e.g.,HS-HS check, HE-HE check, and the like). If the line-level errorhandling module detects at block 640 any violation of the frame syncsignal protocols (YES at block 640), the module flags the current framefor issuing an interrupt to the host processor indicating the protocolerror for the current frame and line (block 642).

In order to ensure downstream modules receive frames and lines with thecorrect size, at block 644, the line-level error handling module furtherchecks if the current line has an illegal line size. At block 644, theline-level error handling module checks the incoming video stream(actual received line size) against reference sizes to detect line widtherrors (e.g., long line, short line) based on the synchronizationmarkers.

If the line-level error handling module detects at block 644 an illegalline size based on the received sync signals and the user configuredreference sizes (YES at block 644), the error handling module goes intoerror handling mode to quickly conceal the line size error and maintainthe line and frame sizes as they are expected by downstream modules.This also prevents subsequent frame loss by avoiding deadlock conditionsin the image processing pipeline. In the line-level error handling mode,the module performs undersized input line handling (block 645A) oroversized input line handling (block 645B), based on whether the linedetected with the illegal line size at block 644 is undersized (e.g.,short line) or oversized (e.g., long line). At block 650, the errorhandling module flags the current frame for issuing an interrupt to thehost processor indicating the line size error. At block 655, the errorhandling module determines if there are more lines in the current inputimage frame, and if yes, begins processing the next line of the currentframe.

As shown in FIG. 6 , operations corresponding to blocks 630-655 areperformed repeatedly and individually for each line of the frame. If, atblock 655, the error handler determines that there are no more linesleft for error handling and concealment processing in the current frame(NO at block 655), method 600 proceeds to block 660 where the interruptis issued to the host processor based on the flag that may be set atblock 642 and/or at block 650 for one or more lines of the current inputframe. Thus, at block 660, the line-level error handling module issuesan interrupt to the host processor to indicate the current frame forwhich (for at least one of the lines of the frame) the line-level errorhandling and concealment operation was performed is a faulty frame. Atblock 665, the line-level error handling module may further transmit tothe host processor (based on the flag that may be set at block 642and/or at block 650 for one or more lines of the current input frame)error reporting information like error location and frame confidencevalue to assist the host processor in determining whether to completelydiscard the frame or use it for subsequent processing based on region ofinterest data corresponding to the frame.

The image processing system with frame-level or line-level errordetection and recovery operations as disclosed herein provides ahardware-based solution to prevent deadlock conditions by maintainingfull frame size and concealing frame errors. The solution providesframe- and line-level handling of frame errors and graceful errorrecovery in hardware (or software or both) to minimize the number oflost frames as compared to the case where a software issued reset isperformed in response to an interrupt for error recovery. Further, inthe line-level error detection and handling operation, error information(e.g., error location information) is provided to the host to minimizeunnecessary frame rejections. By implementing error detection (andconcealment) operation at multiple layers (e.g., on the receiver, on thedownstream image processing unit of the image processing pipeline),protection along the full image data signal path can be ensured.

FIG. 7 shows an illustrative simplified block diagram of computingsystem 700 in accordance with one or more embodiments. Computing system700 may correspond to or may be part of a computer and/or any othercomputing device, such as a workstation, server, mainframe, supercomputer, and/or portable computing device. With reference to FIG. 1 ,computing system 700 may correspond to control unit 105. The computingsystem 700 comprises a processor 702 which may include one or moreprocessors (CPUs, GPUs, or other types of integrated circuits) and/orother type of system on chip (SoC) components to process image data. Asan example, processor 702 may include processing components, such asapplication specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), digital signal processors (DSPs), and memory forperforming various image processing operations associated with the imageor pixel processing pipeline as described herein. The processor 702 maycommunicate (e.g., via a system bus 770) and/or provide instructions toother components within the computing system 700, such as the inputinterface 704, output interface 706, and/or memory 708. In oneembodiment, processor 702 may comprise one or more multi-core processorsand/or memory mediums (e.g., cache memory) that function as buffersand/or storage for data. Although FIG. 7 illustrates that processor 702may be a single processor, processor 702 is not so limited and insteadmay represent multiple processors.

FIG. 7 illustrates that memory 708 may be operatively coupled toprocessor 702. Memory 708 may be a non-transitory medium configured tostore various types of data. For example, memory 708 may include one ormore memory devices that comprise secondary storage, read-only memory(ROM), and/or random-access memory (RAM). The secondary storage istypically comprised of one or more disk drives, optical drives,solid-state drives (SSDs), and/or tape drives and is used fornon-volatile storage of data. In certain instances, the secondarystorage may be used to store overflow data if the allocated RAM is notlarge enough to hold all working data. The secondary storage may also beused to store programs that are loaded into the RAM when such programsare selected for execution. The ROM is used to store instructions andperhaps data that are read during program execution. The ROM is anon-volatile memory device that typically has a small memory capacityrelative to the larger memory capacity of the secondary storage. The RAMis used to store volatile data and perhaps to store instructions.

Memory 708 may be used to house the instructions and logic for carryingout various embodiments described herein. In an embodiment, memory 708may comprise error detection and error handling/concealment logic thatmay be accessed and implemented by processor 702. Additionally, oralternatively, this logic may be stored and accessed within memoryembedded in processor 702 (e.g., cache memory) or implemented inhardware or in some combination of hardware and software. In oneembodiment, memory 708 may interface with system bus 770 (e.g., acomputer bus) to communicate and/or transmit information stored inmemory 708 to processor 702 during execution of software programs, suchas software applications that comprise program code, and/or computerexecutable process steps that incorporate functionality describedherein.

Persons of ordinary skill in the art are aware that software programsmay be developed, encoded, and compiled in a variety of computinglanguages for a variety of software platforms and/or operating systemsand subsequently loaded and executed by processor 702. In oneembodiment, the compiling process of the software program, may transformprogram code written in a programming language to another computerlanguage such that the processor 702 is able to execute the programmingcode. For example, the compiling process of the software program maygenerate an executable program that provides encoded instructions (e.g.,machine code instructions) for processor 702 to accomplish specific,non-generic, particular computing functions, such as performing theerror detection and handling/concealment operations described herein.

After the compiling process, the error detection andhandling/concealment operations described herein may be loaded ascomputer executable instructions or process steps to processor 702 fromstorage (e.g., memory 708, storage medium/media, removable media drive,and/or other storage device) and/or embedded within the processor 702.Processor 702 can execute the stored instructions or process steps toperform instructions or process steps to transform computing system 700into a non-generic, particular, specially programmed machine orapparatus. Stored data, e.g., data stored by a storage device, can beaccessed by processor 702 during the execution of computer executableinstructions or process steps to instruct one or more components withincomputing system 700.

Alternatively, rather than programming and/or loading executableinstructions onto memory 708 and/or processor 702 to form a non-generic,particular machine or apparatus, persons of ordinary skill in the artare aware that stored instructions may be converted and implemented ashardware customized for a particular use (e.g., SOC for ADAS,infotainment, imaging, and computer vision applications). In oneembodiment, implementing operations, such as the error detection andhandling/concealment operations described herein, by loading executablesoftware into a computing device, can be converted to a hardwareimplementation by well-known design rules. For example, the compilingprocess of the software program may build a sequence of instruction bitsthat control and arrange a sequence of control gate-level componentsthat write data onto buses, into latches and registers, across channels,memory, and/or other components of processor 702 and/or memory 708. Thecompiling of the image processing operations may produce gate-levelcomponents with fixed relationships designed to accomplish specific,non-generic, particular computing functions.

The decisions between implementing a concept in software versus hardwaremay depend on a number of design choices that include stability of thedesign and numbers of units to be produced and issues involved intranslating from the software domain to the hardware domain. Often adesign may be developed and tested in a software form and subsequentlytransformed, by well-known design rules, to an equivalent hardwareimplementation in an ASIC or other application specific hardware thathardwires the instructions or process steps of the software. In the samemanner as a machine controlled by a new ASIC is a non-generic,particular, specially programmed machine or apparatus, likewise acomputing device (e.g., a computer) that has been programmed and/orloaded with executable instructions or process steps should be viewed asa non-generic, particular, specially programmed machine or apparatus.

FIG. 7 also illustrates that processor 702 may be operatively coupled toinput interface 704 configured to receive input sensor data and/ordirect-report data, and output interface 706 configured to output and/ordisplay, e.g., image data. The input interface 704 may be configured toobtain input sensor data and/or direct-report data and/or otherinformation via cables, connectors, wireless connections and/or othercommunication protocols. In one embodiment, input interface 704 may be anetwork interface that comprises multiple ports configured to receiveand/or transmit data via a network. In particular, the network interfacemay transmit the data via wired links, wireless link, and/or logicallinks. Other examples of input interface 704 may be universal serial bus(USB) interfaces, CD-ROMs, DVD-ROMs and/or connections to one or moresensors. Output interface 706 may include to one or more connections fora graphic display (e.g., monitors), a printing device that produceshard-copies of the generated results, and/or a plurality of ports thattransmit data via cables, connectors, wireless connections, and/or othercommunication protocols.

FIG. 7 also illustrates that processor 702 may be operatively coupled toone or more device sensors 715. Device sensors 715 may include, but neednot be limited to, an an optical sensor array, a sound sensor, imagesensor, CMOS sensor, an ambient light sensor, a heat sensor, a lightsensor, a differential light sensor, a pixel array, a micro pixel array,and the like. Persons of ordinary skill in the art are aware thatcomputing system 700 may comprise other components well known in theart, such as other sensors, powers sources, and/or analog-to-digitalconverters, not explicitly shown in FIG. 7 .

It is to be understood that the above description is intended to beillustrative, and not restrictive. The material has been presented toenable any person skilled in the art to make and use the claimed subjectmatter as described herein, and is provided in the context of particularembodiments, variations of which will be readily apparent to thoseskilled in the art (e.g., some of the disclosed embodiments may be usedin combination with each other). In addition, some of the describedoperations (image processing methods 400 and 600) may have theirindividual steps performed in an order different from, or in conjunctionwith other steps than presented herein. Also, some of the disclosedsteps may be omitted. More generally, if there is hardware support someoperations described in conjunction with FIGS. 1-6 may be performed inparallel.

Reference in this disclosure to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure, and multiple references to “one embodiment” or “anembodiment” should not be understood as necessarily all referring to thesame embodiment. The terms “a,” “an,” and “the” are not intended torefer to a singular entity unless explicitly so defined, but include thegeneral class of which a specific example may be used for illustration.The use of the terms “a” or “an” may therefore mean any number that isat least one, including “one,” “one or more,” “at least one,” and “oneor more than one.” The term “or” means any of the alternatives and anycombination of the alternatives, including all the alternatives, unlessthe alternatives are explicitly indicated as mutually exclusive. Thephrase “at least one of” when combined with a list of items, means asingle item from the list or any combination of items in the list. Thephrase does not require all of the listed items unless explicitly sodefined.

At least one embodiment is disclosed and variations, combinations,and/or modifications of the implementation(s) and/or features of theimplementation(s) made by a person having ordinary skill in the art arewithin the scope of the disclosure. Alternative implementations thatresult from combining, integrating, and/or omitting features of theimplementation(s) are also within the scope of the disclosure. Wherenumerical ranges or limitations are expressly stated, such expressranges or limitations may be understood to include iterative ranges orlimitations of like magnitude falling within the expressly stated rangesor limitations (e.g., from about 1 to about 10 includes, 2, 3, 4, etc.;greater than 0.10 includes 0.11, 0.12, 0.13, etc.). The use of the term“about” means±10% of the subsequent number, unless otherwise stated.

Many other implementations will be apparent to those of skill in the artupon reviewing the above description. The scope of the disclosuretherefore should be determined with reference to the appended claims,along with the full scope of equivalents to which such claims areentitled. In the appended claims, the terms “including” and “in which”are used as the plain-English equivalents of the respective terms“comprising” and “wherein.”

“The term “couple” is used throughout the specification. The term maycover connections, communications, or signal paths that enable afunctional relationship consistent with the description of the presentdisclosure. For example, if device A generates a signal to controldevice B to perform an action, in a first example device A is coupled todevice B, or in a second example device A is coupled to device B throughintervening component C if intervening component C does notsubstantially alter the functional relationship between device A anddevice B such that device B is controlled by device A via the controlsignal generated by device A.”

Certain terms have been used throughout this description and claims torefer to particular system components. As one skilled in the art willappreciate, different parties may refer to a component by differentnames. This document does not intend to distinguish between componentsthat differ in name but not function. In this disclosure and claims, theterms “including” and “comprising” are used in an open-ended fashion,and thus should be interpreted to mean “including, but not limited to .. . .” Also, the term “couple” or “couples” is intended to mean eitheran indirect or direct wired or wireless connection. Thus, if a firstdevice couples to a second device, that connection may be through adirect connection or through an indirect connection via other devicesand connections. The recitation “based on” is intended to mean “based atleast in part on.” Therefore, if X is based on Y, X may be a function ofY and any number of other factors. The recitation “approximately” beforethe recitation of a value is intended to cover all values within therange of ±10% of the value.

The above discussion is meant to be illustrative of the principles andvarious embodiments of the present disclosure. Numerous variations andmodifications will become apparent to those skilled in the art once theabove disclosure is fully appreciated. It is intended that the followingclaims be interpreted to embrace all such variations and modifications.

What is claimed is:
 1. An image processing device, comprising: an imagedata receiver configured to receive an image data frame, the image datareceiver including an error detector configured to detect whether a sizeof the image data frame is erroneous; an error handler coupled to, andconfigured to receive from, the error detector a current image dataframe, the error handler configured to: perform frame level erroroperations on the current image data frame to generate a corrected imageframe, the frame level error operations including detecting a sizeviolation with respect to the current image data frame based on acomparison of received first, second, third, and fourth signals andexpected first, second, third, and fourth signals, in which: theexpected first signal is active while the error handler receives a firstpixel of an image data frame, the expected second signal is active whilethe error handler receives a last pixel of an image data frame, theexpected third signal is active while the error handler receives a firstpixel of each of a plurality of lines including the first line of animage data frame, and the expected fourth signal is active while theerror handler receives a last pixel of each of the plurality of linesincluding the last line of an image data frame; and an image processorcoupled to receive from the error handler the corrected image dataframe.
 2. The image processing device of claim 1, wherein the errorhandler is configured to detect a size violation with respect to thecurrent image data frame when one or more of the following occurs: thereceived second signal is active while the error handler receives thelast pixel of the current image data frame and the received fourthsignal is not active while the error handler receives the last line ofthe current image data frame, and the received first signal is activewhile the error handler receives the first pixel of the current imagedata frame and the received third signal is not active while the errorhandler receives the first pixel of the first line of the current imagedata frame.
 3. The image processing device of claim 1, wherein the errorhandler is configured to detect a size violation with respect to thecurrent image data frame when at least one of the following occurs: thereceived second signal was not active when the error handler receivedthe last pixel of an immediate previous image data frame, and thereceived first signal is not active while the error handler receives thefirst pixel of the current image data frame.
 4. The image processingdevice of claim 1, wherein the error handler is configured to detect asize violation with respect to the current image data frame when atleast one of the following occurs: the received fourth signal was notactive while the error handler receives a last pixel of each of theplurality of lines including the last line of an image data frame, andthe received fourth signal is not active while the error handlerreceives the first pixel of the first line of the current image dataframe.
 5. The image processing device of claim 1, wherein the errorhandler is configured to detect a size violation with respect to thecurrent image data frame when at least one of the following occurs: thereceived fourth signal was not active while the error handler received alast pixel of a line of the plurality of lines of the current image dataframe, and the received fourth signal is not active while the errorhandler receives a last pixel of a next line immediately following theline of the plurality of lines of the current image data frame.
 6. Theimage processing device of claim 1, wherein the error handler isconfigured to detect a size violation with respect to the current imagedata frame when at least one of the following occurs: the receivedsecond signal was not active while the error handler receives a lastpixel of an immediate previous image data frame, and the received secondsignal is not active while the error handler receives a last pixel ofthe current image data frame.
 7. The image processing device of claim 2,wherein the error handler is further configured to interrupt a hostprocessor in response to detecting a size violation with respect to thecurrent image data frame.
 8. The image processing device of claim 1,wherein, in response to detecting that the frame size of the image dataframe is erroneous, the error detector is further configured todetermine that the image data frame is larger than a reference size, andin response, discard data of a portion of the image data frame togenerate the current image data frame.
 9. The image processing device ofclaim 8, wherein the reference size includes an expected line width. 10.The image processing device of claim 2, wherein in response to detectinga size violation, the error handler is further configured to conceal thesize violation and maintain a frame size expected by the imageprocessor.
 11. The image processing device of claim 10, wherein theerror handler is further configured to perform one of: undersizedprocessing when the error detector detected that the size of the imagedata frame is undersized, and oversized processing when the errordetector detected that the size of the image data frame is oversized.12. The image processing device of claim 11, wherein the error handleris further configured to issue an interrupt to a host processorindicating a frame size error with respect to the image frame data. 13.An image processing device, comprising: an image data receiverconfigured to receive an image data frame, the image data receiverincluding an error detector configured to detect whether a size of theimage data frame is erroneous; and an error handler coupled to, andconfigured to receive from, the error detector a current image dataframe, the error handler configured to perform line level erroroperations on the current image data frame, including fetching data oflines of the current image data frame line-by-line, and for each fetchedline: compare multiple received synchronization signals to expectedsynchronization signals to determine whether at least one of themultiple synchronization signals exhibits an improper transition or isnot received, compare an actual length of the line to a referencelength, and perform line error handling when the actual length of theline is determined to be different than the reference length based onthe compare operation to generate a corrected line, the line errorhandling including concealing the line size error from a downstreamimage processor coupled to the error handler.
 14. The image processingdevice of claim 13, wherein the line level error operations which theerror handler is configured to perform further include: flag the currentimage data frame in response to the error handler determining that atleast one of the multiple synchronization signals exhibits an impropertransition or is not received.
 15. The image processing device of claim14, wherein the line level error operations which the error handler isconfigured to perform further include: flag the current image data framein response to the error handler performing line error handling.
 16. Theimage processing device of claim 15, wherein the error handler isfurther configured to: issue an interrupt to a host processor based onthe flag of the current image data frame, after all of the lines of thecurrent image data frame have been processed.
 17. The image processingdevice of claim 16, wherein the error handler is further configured totransmit to the host processor error information regarding the currentimage data frame.
 18. The image processing device of claim 17, whereinthe error information includes location information indicating whichline or lines of the current image data frame are faulty.
 19. The imageprocessing device of claim 17, wherein the error information includes aconfidence level with respect to the current image data frame.